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ABSTRACT 


An important part of NASA’s mission involves the secondary application of its technologies 
in the public and private sectors. One current application being developed is the The Adult 
Literacy Evaluator, a simulation- based diagnostic tool designed to assess the operant 
literacy abilities of adults having difficulties in learning to read and write. Using ICAT 
system technology in addition to speech recognition, closed-captioned television (CCTV), 
live video and other state-of-the art graphics and storage capabilities, this project attempts to 
overcome the negative effects of adult literacy assessment by allowing the client to interact 
with a intelligent computer system which simulates real-life literacy activities and materials 
and which measures literacy performance in the actual context of its use. The specific 
objectives of the project are as follows: (a) To develop a simulation-based diagnostic tool 
to assess adults’ prior knowledge about reading and writing processes in actual contexts of 
application, (b) to provide a profile of readers’ strengths and weaknesses, and (c) to 
suggest instructional strategies and materials which can be used as a beginning point for 
remediation. In the first and development phase of the project, descriptions of literacy 
events and environments are being written and functional literacy documents analyzed for 
their components. Examples of literacy events and situations being considered included 
interactions with environmental print (e.g., billboards, street signs, commercial marquees, 
storefront logos, etc.), functional literacy materials (e.g., newspapers, magazines, 
telephone books, bills, receipts, etc.) and employment-related communication (i.e., job 
descriptions, application forms, technical manuals, memorandums, newsletters, etc.). 

Each of these situations and materials are being analyzed for its literacy requirements in 
terms of written display (i.e., knowledge of printed forms and conventions), meaning 
demands (i.e., comprehension and word knowledge) and social situation. From these 
descriptions, scripts are being generated which define the interaction between the student, 
an on-screen guide and the simulated literacy environment. The proposed outcome of the 
Evaluator is a diagnostic profile which will present broad classifications of literacy 
behaviors across the major areas of metacognitive abilities, word recognition, vocabulary 
knowledge, comprehension and writing. From these classifications, suggestions for 
materials and strategies for instruction with which to begin corrective action will be made. 
The focus of the Literacy Evaluator will be essentially to provide an expert diagnosis, and 
an interpretation of that assessment which then can be used by a human tutor to further 
design and individualize a remedial program as needed through the use of an authoring 
system. 
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I NTRO DUCT ION 


Secondary Applications of NASA Technology 

As mandated by Congress, one of NASA’s principal charters is to facilitate the 
transfer of its vast technological knowledge into the public and private sectors for the 
general benefit of the American populace. Further, NASA has taken a particular interest 
and lead in promoting educational programs and innovations which not only enrich 
American life, but also serve to maintain and sharpen this country’s scientific and 
technological edge (Haggarty, 1990, Technology Utilization Office, 1991). The Adult 
Literacy Evaluator represents an important spinoff project which will bring to bear NASA’s 
considerable expertise in artificial intelligence (AI) and intelligent computer-aided training 
(ICAT) upon the growing problem of adult illiteracy and diminishing workplace literacy 
skills. This project with its application of expert system technology to the diagnosis and 
evaluation of reading problems represents a significant advance over current computer- 
assisted instruction (CAI) in literacy which provides no intelligent feedback to either 
student or tutor. After a brief overview of some of the issues in adult literacy instruction, 
the following report will describe the current status of the project and its proposed 
development schedule. 

Adult Literacy: A National Issue 

While experts disagree on the exact magnitude of the problem of adult illiteracy both 
in the U.S. and elsewhere, there is no doubt that educators, industry executives and 
government officials are concerned about what seems to be a rising tide of workers who are 
ill-equipped to meet the requirements of an increasingly technological and information- 
based society. Not only docs illiteracy impact a person’s ability to participate fully in a 
democratic society, the cost of remedial education to U.S. corporations each year reaches 
into the billions of dollars annually (Barlow, 1991; Jennings, 1988). This expense coupled 
with the fact that the nation now spends more federal monies on education than on defense 
(U.S. Department of Education, 1991) presents an alarming picture which suggests that our 
current system of public elementary and secondary schooling may have reached a point of 
“diminishing returns” in preparing workers for jobs in an age where information is a 
primary commodity and its rapid and efficient management is of paramount importance. 

In order to improve the quality of education and to ensure that America has a 
competent workforce in the 21st century, the President and the nation’s governors identified 
in 1989 six national goals to be accomplished by the year 2000. In essence, these goals call 
for the participation of business and industry in education, the setting of national standards 
of achievement, and the elimination of illiteracy in the adult population. While the 
feasibility of this latter goal has been questioned (Mikulecky & Drew, 1991), the federal 
government’s efforts to study adult learning and workplace literacy have intensified over 
the last few years with the establishment of the National Center on Adult Literacy at the 
University of Pennsylvania in October 1990 and by the recent passing of the National 
Literacy Act in July 1991 which will provide federal funds for die establishment of state 
literacy resource centers. Further, a new agency, the National Institute for Literacy, is in 
the development stage and will provide additional resources for research and evaluation of 
adult literacy programs (Lovell, 1991). 
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Workplace Literacy Assessment 

Despite these widespread efforts to combat adult illiteracy and increase the quality of 
performance in the workplace, several problems present themselves to researchers and 
practitioners in this field. The first of these concerns the disparity between skills learned in 
school and those actually required on the job. Research has found that job site literacy 
materials not only are of a greater variety than those found in school, but are often more 
difficult. In addition, Mikulecky (1982) has found that in comparison to high school 
students workers actually spend more time reading on the job and perform higher level 
reading tasks such as self-questioning, summarizing or monitoring. Thus, one of the 
serious hindrances to an increased supply of competent workers is that traditional literacy 
instruction in school is very narrow and there is little guarantee that even the skills that are 
learned transfer easily to real-life reading and writing tasks. 

Another area of concern is the lack of assessment instruments which adequately 
measure literacy ability as it manifests itself in real-world reading tasks. Although 
standardized tests exist which are commonly used in evaluating the progress of adult 
education programs, there is widespread consensus (cf. Askov, 1991) that these traditional 
pencil and paper tests lack content validity for indexing job site literacy skills and cannot 
reliably predict a person’s actual reading and writing performance in the workplace. In an 
attempt to remedy this situation, the Department of Labor commissioned the development 
of the Applied Tests of Literacy Skills based upon the 1986 National Assessment of 
Educational Progress survey of young adults, ages 21-25. These tests use facsimile 
productions of newspaper articles, office memorandums job applications, tables, charts, 
etc. to simulate actual workplace materials and tasks. However, it remains to be seen if this 
non-situated, pencil and paper test can overcome the limitations of most psychometric 
measures of its type. 


PROJECT DESCRIPTION AND STATUS 
Project Overview and Objectives 

The scope of the NASA Adult Literacy Evaluator aims at both incorporating real-life 
reading and writing tasks as well as providing through computer simulation actual 
situations encountered in the environment and workplace. The specific objectives of the 
project are to (a) develop a simulation-based, expert system diagnostic tool to assess adults’ 
prior knowledge about literacy processes and materials in the context of their use, (b) to 
provide a profile of readers’ strengths and weaknesses, and (c) to suggest instructional 
strategies and materials which can be used as a beginning point for remediation. The initial 
target population whom the project intends to serve is the two to six percent of the adult 
population whose facility with written language is extremely limited (Elkind, 1990). 
However, one assumption underlying the program is that even adults who’ve failed in 
school or on traditional pencil and paper tests of literacy skills still have considerable 
knowledge about written displays in the environment which are not tapped by present 
methods of testing. A fundamental premise of the project is that unless this prior 
knowledge about print in the environment is known, then corrective instruction will 
inevitably either over-or underestimate the adult’s actual level of ability. In order to create 
these simulated literacy events, the project is in the process of experimenting with the 
following technologies: Closed-captioned television (CCTV), computerized speech 
recognition and production, live video, virtual reality and laser disk storage of video 
images. At present, all of the foregoing are being designed to run on a Macintosh Ilci or 
better with touch screen and voice regulated response to be integrated as is possible. 
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Literacy Knowledge Base 


. Range of components . The first step in developing the Adult Literacy Evaluator has 
been to identify the actual components of literacy that will be simulated and assessed. 
Drawing from the experience of literacy experts, reference materials and the professional 
literature in reading education, over 300 areas of literacy behavior have been categorized 
under six broad rubrics: (a) Aspects of metacognition, (b) word recognition, (c) 
vocabulary, (d) comprehension, (d) fluency, (e) writing, and (0 studying. Under each one 
of these major literacy topics, further subcategorizations have been developed in a logical 
hierarchy, subject to verification as the development process continues. Given the space 
limitations of this paper, it is not possible to display all of the 300 categories and their 
respective definitions. However, represented in Figure 1 below is a subset of components 
arranged under the topic of Written Language Awareness, one of the four subheadings 
under Metalinguistic Awareness which in turn is one of four topics under Aspects of 
Metacognition. Written language awareness is further subdivided into knowledge about 
written conventions, written symbols (graphemes), word boundaries, print orientation and 
direction, sentence boundaries and the relationship between different kinds of pictorial 
displays and the text in which they are embedded. Each of these topics in turn have further 
breakdowns into the specific graphic symbols which appear in written materials and the 
conventional processes or page formats which are used to render these written features into 
the intended meaning of the respective document 

Concepts assessed . Coupled with each component displayed in the tree diagram 
and representing the knowledge base of the system are definitions which outline the 
parameters of what each node means in terms of specific assessment These definitions 
form the basis of the expert system diagnosis and interpretation which will be relevant as 
the subject proceeds through the simulated settings and literacy events as constructed by the 
program. An example of definitions is displayed in Table 1 for the major topic of 
Metacognition, the subheading Metalinguistic Awareness and for some of the components 
under Conventions previously represented (see Fig. 1). Presendy, initial definitions for 
over 100 areas under Metacognition have been written and are currently undergoing 
revision and refinement. Additional descriptions of the knowledge base for the other six 
major literacy areas are yet to be completed. 

Literacy Environments 

Graphic displays . Although most of the estimated 30-60 million adults reading 
below the eighth grade level in this country consistently fail on school-based tests of 
reading and writing, they do not lack knowledge about the written environment and 
oftentimes manage adeptly to either cope with the level of knowledge that they have or to 
leam how to use others’ knowledge for their benefit The intent of the Adult Literacy 
Evaluator is to produce computer-based simulations of everyday and workplace literacy 
situations where print has a ubiquitous presence and give these adults an opportunity to 
display the knowledge that they have in the context in which they apply it. Some of the 
literacy documents intended for use include, but are not limited to, interactions with 
environmental print (e.g., billboards, street signs, commercial marquees, storefront logos), 
functional literacy materials (e.g., newspapers, magazines, telephone books, TV Guide, 
bills receipts, etc.) and employment-related communication (for instance, job descriptions, 
application forms, technical manuals, flyers, newsletters, memorandums, etc.). The range 
of the print in these environments will vary from simple one or two word signs to 
paragraphs. Examples of text to be used in the simulations are depicted in Figure 2. 
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Literacy scenarios . Unlike most testing situations where the knowledge assessed 
is completely out of context, the Literacy Evaluator will take subjects through a series of 
natural literacy activities encountered on common excursions in the environment This type 
of assessment of environmental print knowledge and coping skills is not unlike a case 
study approach to remediation and diagnosis where the clinician or tutor accompanies an 
illiterate adult in a variety of situations and observes how the person negotiates encounters 
with others where written language is normally involved. Three “vignettes” currently being 
developed are a walk through a mall , a grocery store and a tour of NASA itself is being 
considered. Other actual job site situations are planned as well. In each one of these 
scenarios, several types of encounters with print are possible. At each one of these 
encounters, a “script” is being written in which the subject in some instances will converse 
with an on-screen guide who may, as a clinician would, ask a series of questions such as 
“What does this sign tell us?” “What symbols do you recognize?” “Where do you think it 
says, ‘No Parking?”’ In another type of encounter, such as with an ATM machine, the 
person will need to respond to the written screen prompts either by reading them aloud or 
simply pressing the right button in order to complete the task of getting money for some 
further encounter. Finally, in some instances, the subject may be directed to respond by 
using an electronic stylus to fill out an application form, write a check or complete an order 
form for a purchase. The goal of the scenarios developed is to simulate as many situations 
as possible where written language plays an integral role such that an accurate, in- context 
assessment might be made of the adult’s full range of knowledge about and functioning 
with printed forms. 

Graphical user interface and assessment In order to have the widest possible range 
of response capabilities, the project is intending to incorporate along with the standard 
Macintosh interface, speech and handwriting recognition, touch-screen capabilities and 
voice recording. Each response of the adult will be classified, counted or evaluated by the 
appropriate assessment algorithm or AI program software. For instance, all of the print 
used in the scenarios will be classified as to its appearance on lists of basic words and the 
adult’s response to each of these words will be monitored. Additionally, the use of neural 
networks may be used in the speech and handwriting recognition and fuzzy logic 
programming might be appropriate for determining whether a series of behaviors 
constitutes knowledge of a certain literacy domain. The eventual program output to a 
human tutor will involve not only the normal assessment areas such as would be gained by 
the administration of a diagnostic reading test, but also an interpretation of the person’s 
knowledge of and concepts about written language displays as well as their behavior with 
specific print forms. 

It is this latter knowledge, in particular, that is absent from traditional assessments. 
Tests routinely report results in the form of standard scores, grade or age levels or 
descriptive frequencies of words, letters, sentences, etc. that are right or wrong. What test 
results cannot provide is an interpretation of the relationship between scores or the reasons 
why a person responded this way or that. Thus, tutors working with adults with reading 
problems usually operate only on negative information, that is, what they don’t know as 
revealed on non-situated, paper and pencil tests. Therefore, the purpose of the present 
project is to provide both situcued information about a person’s reading ability as well as 
interpretations regarding the person’s performance and what concepts about written 
language would cause the person to respond in such a way. 
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Computing Platform 

The research and development platform of the project is illustrated in Figure 3. 
While an early demonstration of the program made use of the VCR to incorporate live 
video, use of a digital video interface (DVI) board for use with a CD - ROM is faster for 
manipulating video. This technology in turn may give way to a new Apple System product 
call QuickTime, a software extension which would eliminate the need for extra hardware to 
compress video images. In addition, the current speech recognition system needs 
considerable processing power in the form of a workstation, something that would be too 
impractical and expensive in a commercial application. While the Macintosh can use a 
software system with an add-in board called \foice Navigator which allows the user to 
access menu items with voice commands, the system at present must be customized for 
every speaker. Further, the present technology of speech recognition computing systems is 
still at a relatively rudimentary stage, with no system being able to converse easily as would 
a human interlocutor (Smith, 1990). The project plan, therefore, is to incorporate the 
speech recognition component at the level of technology available and use it within 
whatever constraints are necessary. Finally, the intended delivery platform would consist 
of a Macintosh Ilci or better with keyboard and mouse, a headphone and microphone 
setup, a large hard disk or writable CD - ROM and a software system which would handle 
the video features and speech recognition for multiple speakers. During the- development 
phase, all of the above technologies are being evaluated for their utility in enhancing the 
demonstration of literacy concepts as well for their current stage of development and 
anticipated cost. 

Development Schedule 

The overall research plan and schedule specifies three phases for project completion 
given adequate funding: (a) Phase I - Development should be completed by Spring 1992, 

(b) Phase II - Field Test and Refinement would extend through the remainder of the year, 
and (c) Phase III - Commercialization would begin after January 1993. Currently, detailed 
planning with the development phase calls for a complete functional requirements document 
specifying the literacy concepts to be evaluated and the graphical user interface to be 
completed by early Fall of 1991. Following this, a detailed design document implementing 
the functional requirements is to be finished before the end of the 1991. Actual 
programming should then begin during the early months of 1992. Adherence to this 
schedule as mentioned is greatly dependent upon receiving the funding necessary to 
support the additional programmers and media specialists needed as well as to purchase 
equipment and software for development. 


SUMMARY 

The design of this project attempts to minimize the negative effects of past failures 
in school by allowing adults to interact with a sophisticated ICAT system, thus giving 
students some “psychological” distance between themselves and a human evaluator as well 
as providing an opportunity to demonstrate what literacy knowledge they possess in the 
actual context of its use. Further, the use of simulated environmental and workplace 
literacy settings also gives the Literacy Evaluator greater ecological validity than paper and 
pencil tests which measure literacy slolls in isolation only. Finally, the true use of AI 
technology constitutes an advance over current computer-assisted instruction which is 
reduced to only the cataloging of information and which has no knowledge base from 
which to interpret learner characteristics. 
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TABLE 1: DEFINITIONS OF LITERACY COMPONENTS 



Literacy Topics 




O Aspects of Metacognition 


Knowledge of one's own mental processes for acquiring, manipulating, 
and applying information derived from written sources. The ability to 
reflect upon and monitor one's own contents of "mind." 


O Metalinguistic Awareness 


r > 

The ability to treat either oral or written language as an "object 
of thought," understanding that each language system is comprised 
of smaller units which have logical relationships among them- 
selves as well as reference to objects both psychological and 
.physical in the real world. 


O Written-Language Awareness 


> 

The ability to treat written language as an object of thought, 
recognizing that this system represents both oral language 
and meaning at the same time. Knowledge that written 
language is comprised of many smaller units each of which 
.has a variety of invariant as well as relative features. 


O Conventions 


Knowledge that the display of written language for the 
purpose of communication follows certain arbitrary 
rules which vary from language to language. These 
rules relate to all levels of the subunits of printed form. 




O Capital 


Knowledge of the various functions of upper case 
letters such as 

(a) signalling the beginning of sentences, 

(b) designating proper names and adjectives, 

(c) indicating titles or specialized names, 

(d) referring to the Deity, and 

(e) using the pronoun L 
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Figure 2. Examples of Environmental Print 
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Figure 3. Research and Development Platform 















